Smart video digest system and method

ABSTRACT

A method of summarizing events in a video recording includes evaluating at least one video recording to identify event that violate at least one rule. The method further includes excerpting a fragment of the at least one video recording. The fragment contains a depiction of the event. The method also includes causing the fragment to be included in a summary video recording. The rules may relate to a threshold amount of motion in a physical space being recorded in the at least one received video recording, or a threshold duration of motion in a physical space being recorded in the at least one received video recording.

BACKGROUND OF THE INVENTION

Certain types of video recordings, for example surveillance recordingsassociated with security systems, chronicle an environment over longperiods of time. Such video recordings may include only a very fewrelevant events. Finding and presenting those relevant events can be along, tedious process.

Existing video surveillance systems do a poor job of directing areviewer to operational video of interest. The reviewer can search by,for example, date and time, or for video connected with a motion oralarm event, but the reviewer must play back the video one search resultat a time. This is a time-consuming process, so reviewers are unlikelyto conduct reviews unless they already know there is an issue requiringreview.

While some analytic tools aim to filter the search results moreeffectively, the user is still compelled to play back one video at atime.

SUMMARY OF THE INVENTION

The described embodiments automatically create, from one or morereal-time video recordings, a short video that provides a summary ofsignificant events from the real-time recordings.

The following example demonstrates benefit of the described embodiments.Assume that a certain retail establishment includes 10 stores, each with8 surveillance cameras. Assume also that all of the video from these 80cameras is stored on a cloud video storage system, producing 1920 hours(10 stores×8 cameras×24 hours) of video in any given day. A districtmanager in charge of all 10 stores would like to gain useful informationfrom these surveillance videos, but sorting through 1920 hours of video,even with some idea of where to look, is a daunting task.

The described embodiments may automatically create, from one or morereal-time surveillance video recordings, a short video clip (e.g., 10minute; also referred to herein as a highlight reel) that providesrelevant information to the district manager. In certain industries,such as retail chains for quick service restaurants, convenient stores,or food shops associated with gas stations, there are repeated patternsof what the district managers are seeking. The described embodiments mayextract and generalize these questions to create an interface of menuoptions. Using this menu interface, a district manager can specifycertain types of events that he or she is interested in seeing within agiven time period (e.g., within a day, week or month). The describedembodiment may apply various smart video extraction tools to create thehighlight reel based on the district manager's interests.

In one aspect, the described embodiments are a method of summarizingevents in a video recording. The method includes evaluating at least onevideo recording to identify event that violate at least one rule. Themethod further includes excerpting a fragment of the at least one videorecording, the fragment containing a depiction of the event. The methodalso includes causing the fragment to be included in a summary videorecording.

In one embodiment, at least one of the one or more rules relates to athreshold amount of motion in a physical space being recorded in the atleast one received video recording. In another embodiment, at least oneof the one or more rules relates to a threshold duration of motion in aphysical space being recorded in the at least one received videorecording. In an embodiment, at least one of the one or more rulesrelates to presence of a particular color in a physical space beingrecorded in the at least one received video recording. In anotherembodiment, at least one of the one or more rules relates to a quantityof items in a physical space being recorded in the at least one receivedvideo recording. In yet another embodiment, at least one of the one ormore rules relates to a size of an item in a physical space beingrecorded in the at least one received video recording. In anotherembodiment, the one or more rules further includes one or more ruleexceptions.

In one embodiment, at least one fragment in the summary video ispresented at a frame rate other than real-time. Another embodimentincludes limiting the fragment of the first video to a particular periodof time.

One embodiment includes adding additional rules subsequent to initiatingthe evaluating, and continuing the evaluating with the one or more rulesand the additional rules. In another embodiment, at least one receivedvideo recordings includes video feeds from two or more physicallocations.

In another aspect, the described embodiments are a system forsummarizing events in a video recording. The system includes a memorydevice for storing at least one video recording. The system includes amicroprocessor-based workstation configured to evaluate the at least onevideo recording to identify an event that violates at least one rule.The microprocessor-based workstation is further configured to excerpt afragment of at least one video recording, the fragment containing theevent. The microprocessor-based workstation is further configured tocause the fragment to be included in a summary video recording.

In another aspect, the described embodiments are a tangible,non-transitory, computer readable medium for storing computer executableinstructions for summarizing events in a video recording, with thecomputer executable instructions for evaluating at least one videorecording to identify event that violate at least one rule, excerpting afragment of the at least one video recording, the fragment containing adepiction of the event, and causing the fragment to be included in asummary video recording.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing will be apparent from the following more particulardescription of example embodiments of the invention, as illustrated inthe accompanying drawings in which like reference characters refer tothe same parts throughout the different views. The drawings are notnecessarily to scale, emphasis instead being placed upon illustratingembodiments of the present invention.

FIG. 1 illustrates an example of a user interface according to thedescribed embodiment.

FIG. 2 illustrates another version of the user interface of FIG. 1.

FIG. 3 illustrates another version of the user interface of FIG. 1.

FIG. 4 illustrates another version of the user interface of FIG. 1.

FIG. 5 shows an overview presentation of a summary video recordingaccording to the described embodiments.

FIG. 6 shows an example of a dashboard according to the describedembodiments.

DETAILED DESCRIPTION OF THE INVENTION

A description of example embodiments of the invention follows.

The described embodiments provide a summary video recording (alsoreferred to herein as a video digest or video highlight reel), whichincludes one or more video fragments (i.e., short video scenes)excerpted from what is generally a much longer video recording orrecordings, such as one or more real-time surveillance video recordings.The summary video recording is also based on criteria (also referred toherein as rules) provided by a user, i.e., one who is tasked withextracting useful information from video recordings. The particularcriteria will be described in detail herein, but may include forexample:

-   -   (a) Is the video summary fixed duration or variable duration?    -   (b) If fixed duration, what is the video summary duration?    -   (c) Which locations should be monitored?    -   (d) Which areas, within a particular location, should be        monitored?    -   (e) What events, if detected, should trigger excerpting a        fragment to the summary video?    -   (f) Where should the source video recordings be stored?    -   (g) Where should the summary video be stored?

One embodiment may use motion as an event that triggers excerpting afragment to a summary video. The embodiment may use a certain amount ofmotion (e.g., motion across a certain region or motion beyond a certainrate) or lack of motion to trigger excerpting a fragment. Otherembodiments may use other events to trigger excerpting a fragment, suchas detection of color or a lack of color, or size or quantity of anitem.

The described embodiments may provide a user with the ability to setsuch criteria with a user interface, an example of which is shown inFIG. 1. The user interface may be implemented through software executingon any of a variety of processing platforms known in the art. Forexample, the software may implement the user interface on a desktop,laptop or notebook computer, on a smartphone, or on a tablet or othersuch handheld processing device.

The criteria the user sets may be stored locally on the processingdevice that implements the user interface, or they may be pushed to aremote location across a network (e.g., cloud-based storage). Thereal-time video recordings may also be transferred to and stored at aremote location across a network. Accordingly, evaluation of thereal-time video recording against the criteria can be carried out fromanywhere that can access the remote location.

The specific example shown in FIG. 1 depicts the creation of a newsummary video recording. The user interface 100 includes a selectionicon 102 for initiating a new summary video recording. The selectionicon in this case is a graphical button, but other embodiments may useother graphical selection tools known in the art. The user interface 100also includes a name field 104 that allows a user to enter a name forthe new summary video recording. This name field 104 may also displaythe name of a previously-created summary video recording that is openedfor editing on the user interface 100.

The user interface 100 further includes a set of radio buttons 106 forselecting an activity type to be included in the summary videorecording. While radio buttons are shown for this example, othergraphical selection tools may also be used. The example shown in FIG. 1presents three different activity types. One selection is “anyactivity,” which essentially places no restrictions on the activitytype. Another selection allows a user to place a restriction on theduration of activity to be included in the summary video recording.Another selection allows a user to place a restriction on duration oflack of activity to be included on the summary recording.

The user interface 100 also includes selection tools for selectingeither particular locations (stores in this example) or a global “alllocations” choice 108; for selecting specific views at the chosenlocation 110, for selecting evaluation days 112 and times 114; forselecting playback speed for the video fragments excerpted from thelonger video recording and placed on the summary video recording. Theuser interface 100 includes selection tools that allow the user to save118 the selected rules or to delete 120 an existing rule set.

FIG. 2 illustrates a particular set of rules selected by a user. In thisexample, the user selected “all stores” 208 with the “offices” view 210,which encompasses potential activity across all stores in the purview ofthe user (e.g., the district manager). The timeframe selected is thelunch shift from 11:00 am to 1:00 pm, and the type of activity selectedin any activity lasting longer than 30 seconds. This particular set ofrules would be useful to a fast food chain district manager who isconcerned with local managers spending significant amounts of time(i.e., over 30 seconds) in the office, when the local manager should beon the floor handling the lunchtime rush. With this example embodiment,any motion detected in any monitored office, that occurs duringlunchtime triggers a video fragment depicting that motion to beexcerpted from the surveillance video recording and placed into thesummary video recording.

FIG. 3 illustrates a similar rule scenario as is shown in FIG. 2, exceptwith specific stores 308 selected. In this example, only the specifiedactivity occurring in either store 1, store 2, store 6 or store 8 willtrigger a video fragment depicting that activity to be excerpted fromthe surveillance video recording and placed into the summary videorecording.

FIG. 4 illustrates a user interface that may be used by several districtmanagers, so that certain stores 408 within certain districts 409 may beselected by each particular district manager.

One described embodiment may present a summary video overview aftergenerating a summary video recording. An example overview is shown inFIG. 5. In this example, provide an “unusual activity” group 502,depicting results that may have been generated based on rules selectedby a user. Each set of results is tagged with an indication 504 of howmany video fragments are within a particular category. For example, FIG.5 shows that seven fragments were found that violated the rule setconcerning lunch rush. This example further provides a “random” group506 that presents a randomly selected set of video fragments from thesurveillance video recording. This example also provides a summary table508 that provides a listing and description of the fragments presentedfor this overview.

The described embodiments may also provide a dashboard presentation tothe user that provides a high-level view of the summary video recording.FIG. 6 illustrates an example of a dashboard according to the describedembodiments. A first indication 602 provides a green dot with “yes” tothe question, “Did every store open on time,” which indicates that everystore the user is entitled to monitor opened on time. A calendar control604 allows a user to select a particular day for dashboard results. Avideo viewer 606 is provided to show video fragments associated withcertain dashboard items. In this example, a video of the Modesto storeis shown. The video viewer includes a “share clip” (i.e., share videofragment) feature 608 that allows the user to convey the fragment toother users. A “view clip in context” tool 610 allows the user to view asingle camera view, at the date and time on that specific camera in thatselected store. These are all parts of an example dashboard—otherfeatures may alternatively be used or in addition to what is shown.

In some embodiments, the summary video recording may be a fixedduration, selected by the user. With a summary video recording of fixedduration, the duration of each video fragment of the summary video willchange depending upon the total number of video fragments. In someembodiments, the fragment remains displayed in real-time with less ofthe real-time information provided. In other embodiments, the fragmentis “sped up” so that more real-time information is compressed within aparticular amount of time.

It will be apparent that one or more embodiments, described herein, maybe implemented in many different forms of software and hardware.Software code and/or specialized hardware used to implement embodimentsdescribed herein is not limiting of the invention. Thus, the operationand behavior of embodiments were described without reference to thespecific software code and/or specialized hardware—it being understoodthat one would be able to design software and/or hardware to implementthe embodiments based on the description herein

Further, certain embodiments of the invention may be implemented aslogic that performs one or more functions. This logic may behardware-based, software-based, or a combination of hardware-based andsoftware-based. Some or all of the logic may be stored on one or moretangible computer-readable storage media and may includecomputer-executable instructions that may be executed by a controller orprocessor. The computer-executable instructions may include instructionsthat implement one or more embodiments of the invention. The tangiblecomputer-readable storage media may be volatile or non-volatile and mayinclude, for example, flash memories, dynamic memories, removable disks,and non-removable disks. While this invention has been particularlyshown and described with references to example embodiments thereof, itwill be understood by those skilled in the art that various changes inform and details may be made therein without departing from the scope ofthe invention encompassed by the appended claims.

What is claimed is:
 1. A method of summarizing events in a videorecording, comprising: displaying a user interface implemented throughsoftware executing on a processing platform, the user interfaceconfigured to present selection tools for the user to select one or morerules for monitored geographic locations of events within the videorecording to be evaluated, to select one or more rules for monitoredtimes of occurrence of events to be evaluated, to select one or morerules for playback speed for video fragments excerpted from at least onevideo recording and placed on a summary video recording, and to save theselections on a storage as one or more rules comprising at least oneactivity type criterion, at least one monitored geographical location ofevents in the video recording to be evaluated criterion and at least onemonitored time of occurrence of events to be evaluated criterion;generating, by the user interface, one or more rules comprising the atleast one activity type criterion, the at least one monitoredgeographical location of events to be evaluated criterion, and the atleast one monitored time of occurrence of events to be evaluatedcriterion, and saving the one or more generated rules on the storage,wherein the one or more rules relate to a restriction on a duration oflack of activity in a physical space being recorded in at least onereceived video recording; using a microprocessor operatively coupled toa memory device; evaluating, by the microprocessor, the at least onereceived video recording stored on the memory device to identify anevent within the at least one received video recording that satisfiesthe one or more generated rules accessed from the storage, wherein theat least one monitored geographical location of events to be evaluatedcriterion associated with the rules describes a monitored geographicallocation of the event and the at least one monitored time of occurrenceof events to be evaluated criterion associated with the rules describesa monitored time of occurrence of the event that is depicted by videocontent recorded on the at least one received video recording;excerpting, by the microprocessor, a fragment of the at least onereceived video recording, the fragment containing a depiction of theevent, wherein the fragment at least satisfies the at least one activitytype criterion; and causing, by the microprocessor, the fragment to beincluded in the summary video recording.
 2. The method of claim 1,wherein at least one of the one or more rules relates to a thresholdamount of motion in a physical space being recorded in the at least onereceived video recording.
 3. The method of claim 1, wherein at least oneof the one or more rules relates to a threshold duration of motion in aphysical space being recorded in the at least one received videorecording.
 4. The method of claim 1, wherein at least one of the one ormore rules relates to presence of a particular color in a physical spacebeing recorded in the at least one received video recording.
 5. Themethod of claim 1, wherein at least one of the one or more rules relatesto a quantity of items in a physical space being recorded in the atleast one received video recording.
 6. The method of claim 1, wherein atleast one of the one or more rules relates to a size of an item in aphysical space being recorded in the at least one received videorecording.
 7. The method of claim 1, further including (i) causing twoor more video recording fragments to be included in the summaryrecording, (ii) limiting the summary recording to a fixed duration, and(iii) changing an amount of real-time information in each of the two ormore video recording fragments to fit the two or more video recordingfragments within the fixed duration.
 8. The method of claim 1, furtherincluding (i) causing two or more video recording fragments to beincluded in the summary recording, (ii) limiting the summary recordingto a fixed duration, and (iii) speeding up at least one of the two ormore video recording fragments to fit the two or more video recordingfragments within the fixed duration.
 9. The method of claim 1, whereinthe one or more rules further includes one or more rule exceptions. 10.The method of claim 1, further including adding additional rulessubsequent to initiating the evaluating, and continuing the evaluatingwith the one or more rules and the additional rules.
 11. The method ofclaim 1, wherein the at least one received video recordings includesvideo feeds from two or more physical locations.
 12. A system forsummarizing events in a video recording, comprising: a user interfaceimplemented through software executing on a processing platform, theuser interface configured to present selection tools for the user toselect one or more rules for monitored geographic locations of events tobe evaluated, to select one or more rules for monitored times ofoccurrence of events to be evaluated, to select one or more rules forplayback speed for video fragments excerpted from at least one videorecording and placed on a summary video recording, and to save theselections on a storage as one or more rules comprising at least oneactivity type criterion, at least one monitored geographical location ofevents in the video recording to be evaluated criterion and at least onemonitored time of occurrence of events to be evaluated criterion; amemory device configured to store at least one received video recordingand the one or more rules accessed from the storage comprising: the atleast one activity type criterion, the at least one monitoredgeographical location of events to be evaluated criterion, and the atleast one monitored time of occurrence of events to be evaluatedcriterion, wherein the at least one activity type criterion relates to arestriction on a duration of lack of activity a physical space beingrecorded in at least one received video recording; amicroprocessor-based workstation configured to (a) evaluate the at leastone received video recording stored on the memory device to identify anevent within the at least one received video recording that satisfiesthe one or more rules, wherein the at least one monitored geographicallocation of events to be evaluated criterion associated with the rulesdescribes a monitored geographical location of the event and the atleast one monitored time of occurrence of events to be evaluatedcriterion associated with the rules describes a monitored time ofoccurrence of the event that is depicted by video content recorded onthe at least one received video recording, (b) excerpt a fragment of theat least one received video recording, the fragment containing theevent, wherein the fragment at least satisfies the at least one activitytype criterion, and (c) cause the fragment to be included in the summaryvideo recording.
 13. The system of claim 12, wherein at least one of theone or more rules relates to detecting motion in a physical space beingrecorded in the at least one received video recording.
 14. The system ofclaim 12, wherein at least one of the one or more rules relates todetecting a quantity of an item in a physical space being recorded inthe at least one received video recording.
 15. The system of claim 12,wherein at least one of the one or more rules relates to detecting asize of an item in a physical space being recorded in the at least onereceived video recording.
 16. The system of claim 12, wherein at leastone of the one or more rules relates presence of a particular color in aphysical space being recorded in the at least one received videorecording.
 17. The system of claim 12, wherein the at least one receivedvideo recordings includes video feeds from two or more physicallocations.
 18. A tangible, non-transitory, computer readable medium forstoring computer executable instructions for summarizing events in avideo recording, with the computer executable instructions for:displaying a user interface implemented through software executing on aprocessing platform, the user interface configured to present selectiontools for the user to select one or more rules for monitored geographiclocations of events to be evaluated, to select one or more rules formonitored times of occurrence of events to be evaluated, to select oneor more rules for playback speed for video fragments excerpted from atleast one video recording and placed on a summary video recording, andto save the selections on a storage as one or more rules comprising atleast one activity type criterion, at least one monitored geographicallocation of events in the video recording to be evaluated criterion andat least one monitored time of occurrence of events to be evaluatedcriterion; generating, by the user interface, one or more rulescomprising the at least one activity type criterion, the at least onemonitored geographical location of events to be evaluated criterion, andthe at least one monitored time of occurrence of events to be evaluatedcriterion, saving the one or more generated rules on the storage,wherein the one or more rules relate to a restriction on a duration oflack of activity in a physical space being recorded in at least onereceived video recording; using a microprocessor operatively coupled toa memory device; evaluating, by the microprocessor, the at least onereceived video recording stored on the memory device to identify anevent within the at least one received video recording that satisfiesthe one or more generated rules accessed from the storage, wherein theat least one monitored geographical location of events to be evaluatedcriterion associated with the rules describes a monitored geographicallocation of the event and the at least one monitored time of occurrenceof events to be evaluated criterion associated with the rules describesa monitored time of occurrence of the event that is depicted by videocontent recorded on the at least one received video recording;excerpting, by the microprocessor, a fragment of the at least onereceived video recording, the fragment containing a depiction of theevent, wherein the fragment at least satisfies the at least one activitytype criterion; and causing, by the microprocessor, the fragment to beincluded in the summary video recording.
 19. The medium of claim 18,further including instructions for limiting the fragment of the firstvideo to a particular period of time.
 20. The medium of claim 18,further including instructions for transferring the summary video to aremote storage network.